Message Understanding Conference- 6: A Brief History

We have recently completed the sixth in a series of "Message Understanding Conferences" which are designed to promote and evaluate research in information extraction. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe some of the motivations for the new format and briefly discuss some of the results of the evaluations. 1 The M U C Evaluations We have just completed the sixth in a series of Message Understanding Conferences, which have been organized by NRAD, the RDT&E division of the Naval Command, Control and Ocean Surveillance Center (formerly NOSC, the Naval Ocean Systems Center) with the support of DARPA, the Defense Advanced Research Projects Agency. This paper looks briefly at the history of these Conferences and then examines the considerations which led to the structure of MUC-6} The Message Understanding Conferences were initiated by NOSC to assess and to foster research on the automated analysis of mili tary messages containing textual information. Although called "conferences", the distinguishing characteristic of the MUCs are not the conferences themselves, but the evaluations to which participants must submit in order to be permit ted to at tend the conference. For each MUC, participating groups have been given sample messages and instructions on the type of information to be extracted, and have developed a system to process such messages. Then, shortly before the conference, participants are given a set of test messages to be run through their system (without making any changes to the system); the output of each part icipant 's system 1The full proceedings of the conference are to be distributed by Morgan Kaufmann Publishers, San Mateo, California; earlier MUC proeeedings~ for MUC-3, 4, and 5, are also available from Morgan Kaufmann. Beth Sundhe im Naval Command, Control and Ocean Surveillance Center Research, Development, Test and Evaluation Division (NRaD) Code 44208 53140 Gatchell Road San Diego, CMifornia 92152-7420 s u n d h e i m @ p o j k e . n o s c . m i l is then evaluated against a manual ly-prepared answer key. The MUCs are remarkable in part because of the degree to which these evaluations have defined a prograin of research and development. DARPA has a number of information science and technology programs which are driven in large part, by regular evaluations. The MUCs are notable, however, in that they in large par t have shaped the research program in information extraction and brought it to its current s ta te}